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ABSTRACT 


This thesis research combines several proven methods by which public 
participation can be used more effectively in a government decision-making process. 
~The research involved fulfills three primary purposes. First, the research provides a 
flexible user-friendly internet-based platform, whereby the knowledge level of a disparate 
group of stakeholders can be improved with respect to a complex technical subject. 
Second, the research demonstrates a method by which stakeholder consensus is derived. 
Third, the research exhibits a method by which public values are aggregated, whatever 
the level of consensus; the data is then provided to the government for use in a decision- 


making model. 
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I. INTRODUCTION 





A. BACKGROUND 

Since the end of the Cold War, the United States has experienced a significant 
reduction of military forces and bases. The Base Realignment and Closure (BRAC) 
commission was established in 1993 for a single purpose: “to quickly dispose of excess 
activities, as designated for closure or realignment by the U.S. Congress, to local 
communities for reuse and economic revitalization.” (NAVFAC, 1999) The base 
closures not only burden the local celeriac and its people with decreased revenue and 
increased unemployment; but, the closures leave contaminants behind, where the polluted 
lands and waters increase the risk of public health and safety. In 1984, Congress 
established the Defense Environmental Restoration Program (DERP) to evaluate and 
remedy contamination from past Department of Defense (DOD) activities, and to protect 
human health and environment from risks associated with contamination of DOD lands. 
DERP was originally funded by DOD Environmental Account (DERA), whose purpose 
is to provide funding for cleaning at operational installations and formerly used defense 
sites. As a result of BRAC decisions, the number of restoration sites has grown 
dramatically, having doubled since 1989 to approximately 4500 sites today. More 
specifically, the current number of U.S. Navy activities slated for closure is 178 and the 
property at 91 of these activities has been or is slated for disposal. (NAVFAC, 1999) The 
administrative burden for the overwhelming number of closure and cleanup projects has 
grown correspondingly. DOD responded to this growth by devolving DERA into 


separate Army, Navy, and Air Force restoration programs. The DOD now serves as a 
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coordinator for the services, and each service is given the responsibility of managing its 
program more effectively. Breaking the organization into components also allows each 
office to be more responsive to stakeholders of any given project. 

The naval component, known as the Department of the Navy’s Environmental 
Restoration Program (ERP, N) is the responsibility of the Office of the Assistant 
Secretary of the Navy for Installations and Environment (ASN (I&E)). Through the 
office of the Chief of Naval Operations (CNO), the Naval Facilities Engineering 
Command (NAVFAC) manages all shoreside environmental projects and programs, 
providing technical expertise to support the Navy’s environmental initiatives and 
coordinates with legislative and regulatory agencies. NAVFAC also manages a natural 
resource program to enhance the environmental attributes of its land, forests and wildlife. 
(NAVFAC, 1999) Ten engineering field divisions and engineering field activities, 
located across the United States and Europe, provide engineering support and services to 
the naval shore establishment. The Engineering Field Activity, West (EFA, West) serves 
as the NAVFAC regional headquarters for Central and Northern California, as well as 
Nevada. | | 

EFA, West officials face the challenge of choosing remediation technologies that 
satisfy several goals such as reducing cost and decreasing human and ecological risk, 
while satisfying the public. The problem EFA, West faces is to choose the best treatment 
technology using public input. However, EFA, West is not certain as to how to 


operationalize public values. Including public opinion into a decision-making process 


has been a difficult task for government officials because of several barriers that exist for 








utilizing public input such as: geographic distance, logistics cost, available time, and a 
structured model for aggregating input. 
B. PURPOSE 

This thesis research is one of a few which combines several proven methods by 
which public participation can be used more effectively in a government decision-making 
process. The research involved fulfills three primary purposes. First, the er 
provides a flexible user-friendly internet-based platform, whereby the knowledge level a 
a disparate group of stakeholders can be improved with respect to a complex technical 
subject. Second, the research demonstrates a method by which stakeholder consensus is 
derived. Third, the research exhibits a method by which public values are aggregated, 
whatever the level of consensus; the data is then provided to the government for use in a 
decision-making model. 

The method used in this thesis research may be useful to any organization which 
must satisfy a heterogeneous collection of stakeholders with different values and 
interests. The methodology addresses the commonplace problem of having to aggregate 
the public’s values after deriving as much consensus as possible among the participating 
members. The method may help any organization interested in utilizing the concerns of 
stakeholders as an integral part of the decision-making process. 

C; METHODOLOGY 

The methodology used in this thesis research consists of the following: 

® Conduct a literature review of research on various decision-making and 

statistical methods including Multiattribute Utility Theory (MAUT), Value 


Tree Analysis (VTA), Analytical Hierarchy Process (AHP), Delphi 
method, and various aggregation methods. 


Research characteristics of Restoration Advisory Boards (RABs) from the 
following sites: Naval Air Station (NAS) Alameda, Concord Naval 
Weapons Station, Mare Island Naval Shipyard, NAS Moffet Field, 
Treasure Island Naval Station, Treasure Island Naval Station-Hunters 
Point Annex, Oakland Fleet and Industrial Supply Center (FISC). Utilize 
the characteristics to assign the roles to proxy-RAB members. 


Develop a discussion forum for the discussion phase of the Delphi 
method, where stakeholders can voice their opinions and educate others on 
the consequences of various remedy-technology alternatives. Readily 
available and free commercial software is used to develop the forum. 


Ask the proxy-RAB panel to use the forum to express their site-specific 
concems (i. e. values) regarding remedy technologies. In each of two 
Delphi rounds, the panel is asked to rank the set of values in order of 
importance. The level of association between the two rounds is 
determined through the used of simple aggregation methods and 
correlation determination.’. 


Publish the results of the rankings on the web after each round of Delphi 
for further discussion and ranking. 


Determine the proxy-RAB members’ representative final ordinal ranking 
of the set of alternatives using an aggregation technique. 


Convert final ordinal ranking into attribute weights. 


Construct the completed value tree. 


D. JUSTIFICATION FOR THE STUDY 


This thesis research involves the integration of proven decision-making methods, 


namely the Delphi method and VTA using MAUT, into an internet-based platform. 


Traditionally, the Delphi method is performed by sending questionnaires to stakeholders 


(RABs), or organizing a conference at a specific location. This thesis research uses an 


Due to time constraints of research, volunteers, none of them belong to RABs, study the characteristics 
of RABs and subsequently role play for genuine RAB members. | 


: Originally, the Median Ranking method and the Hungarian method were used measure the degree of 
divergence (or convergence) of the proxy-RAB members’ concerns (i.e. values) at the end of each round. 
Efforts to derive an assignment matrix failed due to the data size; and as a result, simpler methods of 
aggregation and correlation were used to legitimate the consensus derived in developing the value tree. 
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internet-based platform to eliminate the logistic barriers such as costs involved with 
administration and logistics and time for all stakeholders to gather. The mean value, 
standard deviation and correlation coefficients are calculated to determine the level of 
association between stakeholders’ opinions, and final consensus. 
E. RESEARCH QUESTIONS 

The primary research question in this thesis research is: 

What is a pragmatic and theoretically sound method to gather the public’s opinion 
into a ngorous decision-making process by using information technology? 

The secondary research questions are: 


® Could this methodology be useful in explaining the many considerations 
in government decisions? 


6 Can the Delphi be implemented on the Internet? 


& How can a technology choice problem be structured by using decision- 
making methods such as Delphi, MAUT, VTA or AHP? 


F. RESEARCH SCOPE AND LIMITATIONS 

The scope of this thesis is to develop a web-based communications structure for 
stakeholders to voice their opinions anonymously, and rank their preferences according to 
their specific values after using the communication structure to learn more about a 
complex and technical subject. Moreover, this research combines the various decision- 
making methods to reach a group’s decision output, which will be given to EFA, West 
officials to decide to adopt the methodology used in this research. 

Although the research provides a valuable demonstration of combining decision- 
making techniques with the Internet, the research has some limitations. First, proxy-RAB 
members substituted for real RAB members due to time constraints. It is impossible to 
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know for certain whether the research results would be different using real RAB 
members. An important factor to consider is that proxy-RAB members do not have a 
genuine and long-held understanding of the concerns that are particular to the actual sites | 
they are representing. Second, time constraint also limited the amount of time for 
discussion during Delphi rounds. Furthermore the number of Delphi rounds is limited to 
two, whereas normally three or more are recommended. 

G. OVERVIEW AND ORGANIZATION 

This thesis is comprised of five chapters. This chapter provides background 
concerning EFA, West’s involvement in the San Francisco Bay Sediment Project. The 
research describes the use of an electronic internet-based forum, where stakeholders can 
voice their opinions and prioritize their concerns vis-a-vis remedy technologies. 

Chapter II is the review of the literature, which explains various decision-making 
and statistical techniques including MAUT, VTA, AHP, the Delphi Technique, the 
Median Ranking Method, the Hungarian Method, and various aggregation methods. 
Chapter JII discusses the formulation and implementation of an internet-based discussion 
forum that allows geographically distant proxy-RABs to list their rmportant concerns, and 
rank the set of attributes anonymously. Chapter IV provides the research results, and 
evaluates the efficiency of the internet, and provides recommendations for developing 
new methodologies to accomplish the purpose of this thesis. 

H. SUMMARY 

For decades, the government has encouraged public participation: however, 

organizations do not have effective models to incorporate public opinion. The Delphi 


method allows the public to participate in a decision-making process, where the public 
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voices its concerns and ranks its attributes anonymously. Delphi has proven itself as a 
systematic and effective way of involving public values. This research investigates 
whether Delphi can be combined with VTA and MAUT using Internet-based elicitation 
techniques. This research creates a bridge that allows government agencies to use such 


methodology in future decision-making practices. 
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Il. LITERATURE REVIEW 


A. INTRODUCTION 

This chapter reviews various types of evaluation and decision-making methods. 
The purpose is to furnish readers with a background of selected decision-making 
methods, which will provide the necessary tools to utilize stakeholder’s values in the 
decision-making process. This chapter is divided into three sections: 1) Multi-criteria 
decision making techniques, 2) Value solicitation and consensus building methods, and 
3) ieee methods. Multi-criteria decision-making techniques, specifically Value 
Tree Analysis (VTA) utilizing Multi-attribute Utility Theory (MAUT), and a competing 
method, Analytic Hierarchy Process (AHP) are investigated, where simply stated 
procedures and applications are provided. Value solicitation and consensus building 
focus on the basics of the Delphi method, the consensus building technique used in this 
thesis. An application of the Delphi using information technology is demonstrated in 
Chapter III. The third section lists various types of aggregation techniques, and explains 
the importance of aggregation when there are various sizes of groups of experts that need 
to make a group decision. 
B. BACKGROUND - ASPECTS OF DECISION-MAKING 

Figure 1 illustrates two major aspects involved in the decision-making process. 
The first is Procedural and Technological Interventions, where well-developed models 
and guidelines are introduced to help facilitate the decision-making process. The second 
aspect encompasses the major features of decision-making, which explains how a 


problem evolves and the need for understanding the problem before making a decision. 
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While the authors give no formal title, this thesis will reference this aspect as Problem 


Structuring and Legitimation. 


Procedural and Technological 


Interventions 


e Procedural Guides 
e Decision Aids 









Problem Structuring and Legitimation 






Problem Context Problem Finding 





















e Social context e Problem 

e Institutional identification 
constraints e Problem acceptance 

e Available e Problem 
information representation 





Problem Solving 





Legitimation Process 




















e Clanfy relevant 
values and beliefs 
e Search for and 


e Determine impacts 
on stakeholders of 
interest/concern 









e Rationalize choice evaluate 
in stakeholders’ alternatives 
terms e Choice 






e Implement choice 





Figure 2.1. Aspects of decision-making (Kleindorfer, Kunreuther and 
Schoemaker, 1993). Note: Changes have been made to original to reflect the relation 
between the two aspects (Procedural and Technological Interventions and Problem 
Structuring and Legitimation). 
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The two aspects explained above are a product of prescriptive and descriptive 
theories based on research by Kleindorfer, Kunreuther and Schoemaker (1993). While 
both theories address the “how” of a decision, descriptive analysis examines the human 
quest for making decisions, while the prescriptive aspect defines measures upon which a 
decision should be based. Consider a group decision situation. An illustration of 
descriptive theory in group decisions would be sociology, which “provides a perspective 
on the nature of information diffusion processes and the role of social norms in the 
decision-making process.” (Kleindorfer, Kunreuther and Schoemaker, 1993, p. 5) Game 
Theory, a prescriptive example, focuses on competition, and is defined as “the study of 
how optimal strategies are formulated in conflict.” (Brown, 1998, p. 31) The following 
paragraphs explain the need for both aspects, and the interdependence that must exist for 
a sound decision-making process. 

1. Procedural and Technological Interventions 

Procedural and Technological Interventions provide structured guidelines for 
individuals or groups who must make a decision. In operations analysis, models are 
developed to answer the well-known economic tradeoff problem of maximizing profit 
while reducing risk. Technology plays a significant role by introducing systems that help 
decision-makers evaluate complex problems. 

Guidelines and models offer prescriptive solutions to a decision-maker; but at the 
same time, lack the human intervention that must be considered. The need for conflict 
resolution among decision-makers with different goals and objectives helps define the 
organization’s structure and decision-making process. This void of the human psyche 
warrants the need for the descriptive aspect of evaluating complex problems. 
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2. Problem Structuring and Legitimation 

In Figure 1, Problem Structuring and Legitimation describe some features 
involved in descriptive theory and is broken into four major categories: Problem context, © 
problem finding, =e solving and legitimation process. (Kleindorfer, Kunreuther and 
Schoemaker, 1993) Problem context defines the social, institutional and informational 
environment of the decision process. Understanding problem context means recognizing 
who might be affected, who has power (or final authority), what information is available 
to the decision-makers, and if possible, what values or beliefs they hold regarding the 
problem in question. 

Problem finding addresses the fact that a problem has been identified by the 
activity, accepted by the decision-maker(s), and represented to give them a decision 
opportunity. Problem solving involves making a choice that resolves the presented 
problem. This choice is based on understanding the relevant values and identifying 
solution alternatives. The Legitimation process consists of determining how the 
stakeholders will be affected by the chosen solution alternative, and rationalizing the 
choice of the decision-maker in terms of stakeholder values and interests. (Kleindorfer, 
Kunreuther and Schoemaker, 1993) 

The upward arrow between Procedural and Technological Interventions and 
Problem Structuring and Legitimation (see Figure 1) represents the need for a problem to 
be well defined and understood by the decision-makers before utilizing a decision- 
making process. Kleindorfer, Kunreuther and Schoemaker state that “sound prescription 
must begin with good description.” (1993, p. 10) The arrow’s downward direction 


indicates that the use of specific types of Procedural and Technological Interventions may 
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help the decision-maker better understand and define the problem. If the desired outcome 
is not obtained, it 1s because of ill-defined factors that affect the overall problem. The 
arrows connecting the categories in the second aspect indicate there are no set procedures 
of which category should be resolved first. Rather, it is a redundant or cyclical process, 
where one change in one category affects the other three iia Only when the 
decision-maker understands all four categories in the aggregate can the problem be 
considered well defined, and only then can the proper tools be selected or designed to 
help the decision-maker make decisions. | 
C. MULTI-CRITERIA DECISION MAKING TECHNIQUES 

Questions like “Is this plan wise?” “Should I choose option A or option B?” 
“How well is this program doing?” have been asked about social programs for decades 
(Edwards and Newman, 1982). But the idea that one could answer such questions 
systematically and in a manner other than by simply making an intuitive judgment, is a 
product of decision science research since the 1950s and 1960s. Organizations strive to 
work more efficiently by choosing programs worthy of support over those that are not, 
and providing guidance for programs in existence. At the same time, organizations 
devote resources and attention to develop satisfying methods to improve efficiency in 
programs. 

i Multi-Attribute Utility Theory (MAUT) 

a. Definition of MAUT 
Multi-attnbute Utility Theory (MAUT) is a tool used to structure a 

decision-maker’s preferences. MAUT presents the approach to determine the priorities 
of the decision-maker, and is a tool in which organizations can organize and aggregate its 
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evaluative efforts. MAUT applies to situations with uncertainty and multiple, often 
conflicting objectives (Murphy, 1997). According to Edwards, Guttentag, and Snapper 
(1975), MAUT depends on a few key ideas: 

@ When possible, evaluations should be comparative. 


® Programs normally serve multiple constituencies (those that have the 
power to vote/elect). 


® Programs normally have multiple goals, not all equally important. 

® Judgments are inevitably a part of any evaluation. 

@ Judgements of magnitude are best when made numerically. 

® Evaluations typically are, or at least should be, relevant to decisions. 


Evaluations of programs that serve multiple constituencies should 
normally be addressed to the interests of those constituencies. Different constituencies 
can be expected to have different interests. If programs have multiple goals, evaluations 
should assess how well the programs serve the constituencies and the programmatic 
goals; this implies multiple measures and comparisons. Edwards and Newman state that 
using “multiple measures of effectiveness (which may well be simple subjective 
judgments in numerical form) makes less appealing the notion of social programs as 
experiments.” (1982, p. 8) However, implementation of such experiments become more 
difficult as the number of measures needed for a satisfactory evaluation increases. 
Experimental or hard data can be easily incorporated in a MAUT evaluation, provided 
such data are available. 

Edwards and Newman (1982) suggest that combining subjectivity with 


evaluation with numerical judgements serve several useful purposes: 
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e First, the combination partly closes the gap between intuitive and 
judgmental evaluations, and makes coexistence of judgement and 
objective measurement within the same evaluation easy and natural. 

® second, accepting subjectivity into evaluation provides an opportunity to 
combine links of values. For instance, “evaluation researchers often 
distinguish between process evaluations and outcome evaluations.” 
(Edwards and Newman, 1982, p. 9) Process and outcome are different, 
but if a program has goals of both kinds, its evaluation can and should 
assess its performance on both. 

® Third, the use of subjective inputs can reduce the time required for an. 
evaluation to be executed. In as little as a week of concentrated effort, a 
MAUT evaluation can be used from its original definition of the 
evaluation problem to the preparation of the evaluation report. 

b. Purposes for Evaluations 

According to Edwards and Newman, MAUT is a “widely applicable 
method of organizing and presenting evaluative information. As such, it is compatible 
with any other evaluative activity designed to yield numbers as outputs.” (1982, p. 11) 
The organizational requirement for evaluation is normally based on the supposition that 
decisions need to be made. Sometimes, the decisions involve programmatic continuation, 
modification or elimination. Other decisions might revolve around program design, 
management, or process improvement. 

Major evaluations are often required as a basis for potential major 
programmatic changes — up to and including the most major of all changes: the birth or 
death of a program. Funding-level decisions are also programmatic choices; the same 
program at two substantially different funding levels results in qualitatively different 
programs. 

The aforementioned decisions share two characteristics. The first is that 


all evaluations require comparison of something with something else. An important 
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implication of the comparative nature of virtually every evaluation is that some of the 
comparisons are inevitably between the program as it is and the program as it might be — 
that is, between the existing and the conceptual programs or programmatic methods. The 
second is that programs virtually always have multiple objectives; consequently, 
evaluations should assess as many objectives as possible. MAUT serves as a useful tool 
a these cases because, as mentioned before, MAUT is directed at ee problems 
with multiple goals serving multiple constituencies. 
c. Steps in a MAUT Evaluation 
In many decision theory books, authors attempt to define the pertinent 
steps that are used to execute the MAUT evaluation. Some have stated five steps, others 
more. To define each step clearly, the process is broken into seven steps, as used by 
Edwards and Newman (1982): 
® Step one: Identify the objects of evaluation and the function or functions 
that the evaluation is intended to perform. The functions of the evaluation 
will often control the choice of objects of evaluation. 
& Step two: Identify the stakeholders. Stakeholders are defined as “people 
who have an interest or stake in the program or entity being evaluated, and 


who are important enough so that their interests should be considered.” 
(Edwards and Newman, 1982, p. 17) 


e Step three: Elicit the relevant value dimensions or attributes from 
stakeholder representatives, and organize the value dimensions or 
attributes into a hierarchical structure. 


® Step four: Assess the relative importance of each of the values identified 
in step three for each stakeholder group. Such judgements can be 
expected to vary from one stakeholder group to another; methods of 
addressing such value conflicts are important. 


® Step five: Ascertain how well each object of evaluation (i.e. alternative) 
meets the values implicit in each attribute. To understand the level of any 
alternative’s success in meeting an attribute requires measurements or 


é 
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subjective expert judgements, or both. These measures and judgements 
must be converted into a common, comparable scale, called utility. 


Step six: Aggregate utilities with measures of importance. This is done 
with the general equation: 


n 
Uj a LWwiuij 
1 = 


Where U; is the overall or composite utility for the j" decision alternative; 
w; is the normalized weight assigned to the i™ attribute; and uj 1s the utility 
of the j" option on the i™ attribute. The symbol x means to sum the 
weighted utilities over all the attributes from the first (1) to the last (n). 


The larger the numerical value of U, the better. Therefore, whatever 


‘decision alternative received the largest U should be considered best 


among the alternatives. 


Step seven: Perform sensitivity analyses. The question underlying any 
sensitivity analysis is whether a change in the analysis will lead to 
different conclusions, for example, using different weights in the utility 
measurement. While conclusions may have emerged from step six, the 
conclusions deserve credence as a basis for action only after their 
sensitivity 1s explored in this step. 


d. The Relation Between Evaluation and Decision 


Researchers have been concerned with the relationship between evaluation 


and decision, especially the academic evaluation researchers, who wonder whether or not 


their evaluations are used appropriately. Some evaluators believe their responsibility is 


to provide the relevant facts; and someone else should make the decisions. As a rule, the 


evaluator is not the decision-maker, and cannot compel the decision-maker to agree with 


the result of the evaluation, or to base decisions on it. 


Edwards and Newman (1982) explain three devices that make evaluations 


more likely to be used in decision-making. The first and most important, is to involve the 


decision-makers heavily in the evaluative process; especially when the decision-makers 
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are among the most important stakeholders. The second is to make the evaluation as 
directly relevant to the decision as possible, preferably by making sure that the options 
available to the decision-maker are the objects of evaluation. The third is to make the 
product of the evaluation useful — which primarily means making it understandable. In 
this research, the proxy-RAB members (representing stakeholders) identify the 
parameters on which evaluators should focus. Additionally, the decision-makers use the 
stakeholder-defined value structure to make decisions. 

e. Summary 

This research conforms to the recommendations of Edwards and Newman 
by ensuring that decision-makers are involved in the evaluation process; by making 
evaluation as directly relevant to the decision as possible; and by making the product of 
the evaluation understandable. The latter point is particularly important in a publicly 
funded program that depends upon constituent satisfaction. Satisfaction, in turn, can only 
be produced by legitimating the decisions. In this spirit, the method adheres to the 
requirements needed to use the simple additive utility as depicted in Section C.1(c). The 
requirements for using the Additive Utility Function, and a further description of this 
method, are included in Appendix A. 

2. Value Tree Analysis (VTA) 

Complex evaluation problems can be structured by building a value tree 
(Edwards, 1980). Value trees are pyramid-like structures. At the top of the tree is 
generally one overall value. “Values are abstractions that help organize and guide 
preferences. They are most often expressed as statements of desired states, positive 


intentions, or preferred directions. The actions or objects of value may be such diverse 
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choices as social policies, marketing strategies, or individual consumer choices.” 
(Winterfeldt and Edwards, 1986, p. 38) Values are also defined as a collection of 
objectives. Clemen states, “An individual’s objectives taken together make up his or her 
values. They define what is important to that person in making a decision. A person’s 
values are the reason for making decisions in the first place. Without objectives, it would 
not be possible to tell which alternative would be the best choice.” (1996, p. 20) 

Below the overall goal are several intermediate level goals, or objectives, needed 
to reach the overall goal. Intermediate goals can include fundamental and means 
objectives. Distinguishing between fundamental and the means objectives allows the 
decision-maker to structure the value tree. “Fundamental objectives reflect what we 
really want to accomplish and means objectives help achieve other objectives. 
Structuring the fundamental objective hierarchy is crucial for developing a multiple 
objective decision model.” (Clemen, 1995 p. 44) 

At the lowest level are the attributes, “which serve to explicate and operationalize 
higher level values and concerns.” (Stillwell, Winterfeldt, and John, 1981 p. 1) An 
attribute is defined as the criteria for measuring the objectives. Rowe (1992) explains an 
attribute as the quantity measured on an attribute scale. An example of an attribute scale 
is the dollar amount if the objective is to minimize cost. Figure 2 is a simple example of 
a value tree. 

According to Winderfeldt and Edwards, the theory of the use of value trees for 
evaluating options is straightforward: “the analyst simply uses the operational attributes, 
obtains estimates of how the options perform on these attributes, converts these estimates 
to utilities, weights the attributes, and carries out the appropriate calculations to generate 
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an overall evaluation of the options.” (Winderfeldt and Edwards, 1986, p. 45) However, 
in practice this approach has difficulties: 
e There may be too many attributes to carry out an evaluation. 


e Some branches of the value tree (see Figure 2) may be irrelevant because 
some of the attributes might not have effect on the solution alternatives. 


® The decision-maker may find higher level branches of the tree to be more 
meaningful than the attributes itself. The task of structuring relevant and 

meaningful operational attributes can be difficult. 
Despite the difficulties mentioned above, a value tree is a useful tool. Winderfeldt and 
Edwards suggest that, “value trees are most often used as background material in the 


development of attribute sets for a formal evaluation.” (Winderfeldt and Edwards, 1986, 


p. 45) 


Overall goal Intermediate goals Measurable attributes 
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To choose the Efficiency Total Cost 
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. The number of complaints 
Reduced nisk The pollutants in atmosphere 
branch The degree of public involvement 





Figure 2.2. Value Tree Example. 
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a. Purpose of Using Value Trees 

Winderfeldt and Edwards (1986) explain three purposes for using value 
trees. First, value trees facilitate the process of ensuring that the final set of attributes 
actually captures all the relevant and useful values. Second, value trees identify the 
appropriate level of abstraction at which it is useful to carry out the evaluation. And 
third, value trees communicate concisely the entire spectrum of values and concerns of 
the decision-maker about the issue at hand. 

b. Criteria for Evaluating Value Trees 

Keeney and Raiffa (1976) propose the following criteria for evaluating the 
objectives and the attributes in a value tree: completeness, operationality, 
decomposability, absence of redundancy, and minimum size. “Completeness requires 
_ that all relevant values be included in the superstructure of the tree, and the substructure 
completely defines the higher level values. Operationality requires that the lowest level 
values or attributes be meaningful and assessable. Decomposability means that the 
attributes can be analyzed one or two at a time, that is, that they are judgmentally 
independent. Absence of redundancy means that no two attributes or values mean the 
same thing. Minimum size requirement refers to the necessity of keeping the number of 
attributes small enough to manage.” (Winderfeldt and Edwards, 1986, p. 43) However, 
the authors have found that these requirements do not always work together. The authors 
say, “These requirements conflict. Operationality often requires further decomposition, 
thus increasing the number of attributes. Completeness may lead to redundancy, since 
true value independence is often an unattainable ideal.” (Winderfeldt and Edwards, 1986, 
p. 43) 


21 


3. Analytical Hierarchy Process (AHP) 

In the 1970s, Thomas L. Saaty introduced an alternate method of solving multiple 
criteria problems called Analytic Hierarchy Process (AHP). AHP helps to solve the same 
problems for which MAUT is used: setting priorities, generating a set of alternatives, 
making decisions using benefits and costs, predicting outcomes (including risk 
assessment), measuring performance, ensuring system stability, and conflict esciitions 

Thomas Saaty and Luis Vargas report that in decision-making, people suai 
subjective judgments based on feelings and intuition rather than on well-formulated 
logical reasoning (1991). “AHP should be used for problems involving a large number of 
attributes whose outcomes can only be measured on a relative scale.” (Tiley, 1994, p. 27) 
The Analytical Hierarchy Process (AHP) demonstrates a representation of the mental 
process, by which overall judgments come about in situations involving complex goals 
and criteria. In “Prediction, Projection and Forecasting,” (1991) Saaty and Vargas define 
AHP as a multi-objective, multi-criteria decision-making approach, which employs a 
pair-wise comparison procedure to arrive at a scale of preferences among sets of 
alternatives. The authors add that to apply the AHP approach, it is necessary to break 
down a complex unstructured problem into its component parts. The parts, or variables, 
are arranged into a hierarchic order; assign numerical values to subjective judgments on 
the relative importance of each variable; and synthesize the judgments to determine 
which variables have the pence priority and should be acted upon to influence the 
outcome. 

Saaty and Vargas continue by stating AHP is “intimately connected with the idea 


of consistency of thought. It [the process] admits inconsistency (including lack of 
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transitivity) and measures the effect of different levels of consistency on the results” 
(1991, p. 12) that are sought. Also, AHP “avoids aprioristic assumptions and models, 
and calls on the individual seeking to know, to construct the framework of that quest.” 
(Saaty and Vargas, 1991, p. 12) The assumption in this thesis is that, judgment is based 
on the acquired experience, reason, and intuition. The objective of AHP is to “fulfill the 
purposes of the people concerned as well as possible, rather than to legislate to them that 
the outcome should be based on principles set for by other people, who are essentially 
outsiders to the problem. Thus, perceived constraints must be examined and not taken for 
granted — the only hope there is to plan a way out of difficult problems.” (Saaty and 
Vargas, 1991, p. 12) 

The AHP synthesizes two approaches by identifying the outcomes that are more 
beneficial to the actors, and at the same time, by providing a way of assessing the factors 
(causes) which may have more to do with certain types of outcomes. 

In using AHP, objectives must be determined first: what needs to be done? 
Second, generate alternatives to satisfy those requirements: what are the possible ways of 
action? Third, set priorities according to the importance of the requirements that allow 
the implementation of the alternatives to attain some higher objective. Finally, choose 
the best policy alternative, or as in some cases, a mix of the best policy alternatives. The 
alternatives depend on the overall obj ective at hand. The choices may be either to 
allocate resources, assess the risk benefits and costs of the allocation, predict future 
consequences; or, measure the level of performance of the system, and possibly redesign 
the overall structure to fit the functions that must be performed to meet certain 
requirements. If the desire is to change the structure of a system, specific goals should be 
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achieved by implementation of specific policies. AHP can assist in structuring problems 
in a system, thereby assisting to prioritize the decision-maker’s objectives. 

AHP also has its pitfalls, such as rank reversal. Rank reversal refers to the 
concept of the alternative rankings that changes when new information is introduced into 
the problem (1994). The mathematical algorithms used in AHP may also be difficult to 
explain to those who are not mathematically inclined. In this thesis research, the need for 
legitimation precludes the use of AHP in favor of MAUT. 

D. SOLICITATION AND CONSENSUS BUILDING 
1. Delphi Method 

a. Background 

For the purposes of this research, there is a need not only to elicit public 
values, but also to have a kind of consensus, a method to summarize the elicited values of 
several individuals. The Delphi technique is a method to meet these needs. Linstone and 
Turoff define the Delphi technique as “a method for structuring a group communication 
process so that the process is effective in allowing a group of individuals, as a whole, to 
deal with a complex problem.” (1975, p. 3) | 

The Delphi technique was developed during the 1950s by two researchers 

at the RAND Corporation as a procedure to ‘obtain the most reliable 

consensus of opinion of a group of experts ...by a series of intensive 

questionnaires with controlled opinion feedback.’ (Dalkey & Helmer, 

1963) Initially, it [the Delphi] was used in the realm of long-term 

forecasts of change, particularly in science and technology. (Rowe, 1992) 


In later years, many researchers have employed the Delphi technique in a variety 


of fields including determining the educational policies, predicting the future 
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technological problems, effects on society, and determining the future issues of real 
estate. 

According to Rowe (1992), Delphi has four necessary characteristics: 
anonymity, iteration, controlled feedback and statistical aggregation of group response. 
Anonymity is achieved through the use of questionnaires which allow participants to 
respond privately. Anonymity reduces the influence of dominant individuals. Iteration 
of is the process of presenting the questionnaire over a number of rounds, which 
encourages respondents to change their minds. Controlled feedback takes place between 
rounds, in which each respondent 1s presented with the opinions of other respondents in 
the form of a statistical summary. At the end of the Delphi procedure, a statistical group 
response 1S obtained that illustrates the spread of opinion reflecting the degree of 
consensus on the issue. 

b. The Delphi Process 

The Delphi process involves seven distinct tasks as shown in Figure 3. 
Creating the panels is the first critical task. Linstone and Turoff (1975) consider three 
kinds of panelists as ingredients for creating a successful mix: stakeholders; those 7 
are or will be affected, experts, those who have an applicable specialty or relevant 
experience; and facilitators, those who have skills in clarifying, organizing, synthesizing, 
stimulating individuals who can supply alternative global views of the culture or society. 
Judging the desirability of individuals considered for the respondent group is subjective 
and therefore, according to some critics, is subject to mistake. However, subjectivity is a 


problem when formulating any group activity- committees, study groups, etc. The 


2D 


problem is not unique to Delphi, and does not detract from the method usefulness. 


(Linstone and Turoff, 1975) 


Expert Panel Selection 
Statistical Analysis of First Round Responses 


Second Round Questionnaire Distributed 
Together with Results of First Round and 


Supporting Information from Experts 





First Round Questionnaire 
Distributed 
Statistical Analysis of Second Round Responses 
Third and Further Rounds as Needed 


Figure 2.3. Delphi method (Kleindorfer, Kunreuther and Schoemaker, 1993 p. 234). 


A questionnaire, prepared and distributed to participants, asks participants 
to provide their opinions that they see as being important. In this research, the 
respondents are asked to list the most important issues regarding the technology choice 
problem. “After the responses from the members are collected and all the redundant data 
is eliminated, the resporises are then summarized statistically. At the end of round one, a 


second-round questionnaire is organized, which is entirely similar to the first-round . 
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questionnaire, except that a statistical summary of first-round responses from the panel is 
also enclosed.” (Kleindorfer, Kunreuther and Schoemaker, 1993, p. 236) 

The second round questionnaire usually mcludes instructions, a statistical 
summary of first-round responses, and a section where the participants can provide any 
additional comments. Each summary statement has a corresponding scale so that each 
participant will rate all the statements. 

Although this research uses an ordinal scale to rate statements, many 
Delphi research efforts make use of the Likert Scale, which is described below. The 
scale can have as the lower anchor, onal disagree (SD) —2; and the higher anchor, 
strongly agree (SA) +2. Neutral (N) is assigned a zero value, disagree (D) and agree (A), 
-1 and +1, respectively. The zero point provides a neutral option to panel members and is » 
intended to overcome forced-choice responding and provide members every possible 
option in rating. A threshold value can be set at the end such as statements rated below +1 
rated statements will not be included in the next round. The purpose of the threshold 
value is to eliminate the extreme ideas. The responses of the participants at the end of 
this round are analyzed by utilizing the descriptive statistics. (Clayton, 1997) 

In the third round, participants are provided the summarized data that is 
obtained at the end of round two. Group mean and individual ratings are provided to the | 
participants so that each participant can see where his or her responses fall with respect to 
the measure of central tendency. Each member is then asked to re-rate the statements 
having the knowledge of opinions expressed in the previous round. Participants are also 


given the opportunity to provide their reasons to their ratings. The responses are 
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summarized again, and the participants are asked to re-rate the statements in the light of 
the new summary. 

This process goes on until either a desired level of consensus is reached, or 
until a time constraint or other constraint is reached. Different authors agree that a 
successful Delphi can be carried out within a three-round format. “In practice, most 
Delphis on policy try to maintain a three- or four- round limit by utilizing the following 
procedure: (1) the monitor team devoting a sensiderble amount of time to carefully 
preformulating the obvious issues; (2) seeding the list with an initial range of options but 
allowing for the respondents to add to the lists; (3) asking for positions on an item and 
underlying assumptions in the first round.” (Linstone and Turoff, 1975, p.88) Usually, 
the Delphi process stops when most of the responses indicate a consensus, or little change 
with further rounds is expected in the distribution of responses from participants. This 
research will conduct two rounds because of time constraints. 

The above described Delphi methodology is called conventional Delphi. 
There are two other types of Delphi methods, real and policy. “Real-time Delphi differs 
from conventional Delphi in that, rather than taking weeks to conduct the process, it 
occurs during the course of a meeting or conference.” (Clayton, 1997, p. 377) In the real- 
time Delphi, the monitor team is replaced by a computer, which has been programmed to 
carry out the compilation of the group results. This approach has the advantage of 
eliminating the delay caused in summarizing each round of Delphi, thereby turning the 
process into a real-time communication system.” (Linstone and Turoff, 1975, p.5) 

The Policy Delphi is a technique used to achieve “every possible opposing 


view” (Linstone and Turoff, 1975, p.84) on an issue. According to Clayton (1997), in 


28 








Policy Delphi, the decision-maker is not interested in having a group generate his 
decision, but rather in having and informed group present all the options and supporting 
evidence for his/her consideration. The policy Delphi is not a mechanism for making 
decisions as generating a consensus is not the prime objective. “A policy Delphi should 
_ be able to serve one or any combination of the following objectives: (1) to ensure that all 
possible options have been put on the table for consideration; (2) to estimate the effect of 
any particular option; (3) to examine and estimate the acceptability of each option.” 
(Linstone and Turoff, 1975, p. 84) 

The major strengths of Delphi are summarized as: Delphi technique helps 
to establish convergence of ideas over an issue. In Delphi, questionnaires can be sent to 
participants by mail or other means, eliminating the need to travel long distances to get 
together at the same place and time. Therefore, Delphi is cost effective. During 
implementation of Delphi, participants are required to provide their responses to the 
questionnaires privately. This process eliminates the negative effects of dominant 
groups, rank, personality, etc. 

Linstone and Turoff (1975) identified inherent difficulties in the Delphi 
method. These include technical difficulties of re-adapting the method for each issue, the 
arbitrariness of panel members (respondents) selection, the balancing of closed and open- 
ended responses in the questionnaire, and estimating the time required for completion. 
At the end of each round, controlled feedback is provided to the respondents. In the 
process of feedback, limiting or eliminating some of the alternatives might have negative 


effects on the success of Delphi. 
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The number of respondents depends on the type, purpose and the 
complexity of the subject. According to Linstone and Turoff (1975), real-time Delphi 
and conventional Delphi can be given from a small to a large group and a policy Delphi 
can be given to a group ranging from ten to fifty people. This research is categorized as 
Policy Delphi, but utilizes fewer respondents than Linstone and Turoff suggested. 

E. AGGREGATION METHODS 

Aggregation is used to measure the level of consensus derived from the Delphi 
method. Aggregation is defined as collecting inputs into a mass, yielding a collective 
output. It has been studied in mathematics, and has expanded into the decision sciences. 
When individual opinions are aggregated, the result 1s a single opinion, representing a 
group choice. Examples of basic aggregation methods include, mean, median, mode and 
Nominal Group Technique. Advanced methods presented in this section include the 
Hungarian method and Median Ranking method. 

i: Basic Methods of Aggregation 

Consider the following example of several Restoration Advisory Board (RAB) 
members listing various attributes in a Delphi forum. The panelists are comprised of 
seven different RAB enbeee each from a different location. The panelists define the 
attributes during the Delphi forum, and rank them in order of importance according to 


their respective RABs. No ties among the attributes are allowed (see Figure 4). 
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Mean, also known as average, is the most common measure of location for a set 
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Figure 2.4. Example RAB Attribute Table. 










of numbers. According to Wheeler and Chambers, average identifies the center of mass 
for the values in the data set (1992). Therefore, it can be considered the balance point 
among the data. The calculation of mean is fairly stmple. It is the sum of all the values 
(In this example, the number represents the ordinal ranking of attributes, showing the 
relative importance of each attribute, i.e. 5 means 5"" most important, 3 means 3 most 
important, etc.) divided by the total number of values. Consider the above example: the 
total score of Attribute 1 is 28 (the sum of 5, 2, 3, 4, I, 6, 7). The mean is the sum of 
these seven values, divided by 7 (28/7 = 4). This value is the balance Séink for Attribute 
1, and the mean values for Attributes 2 through 7 are presented in Figure 6. Therefore, if 
the RAB members decided the lowest number was the most desired, then Attribute 5 


would be the best decision. 
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A median is another measure of location for a data set. This statistic identifies the 
50" percentile of the data, indicating at least half the data are greater than, or equal to the 
median; and at least half the data are less than, or equal to the median. Therefore, while 
mean describes the balance point of the data set, median splits the data set into two equal 
halves. In calculating the median, the first step is to arrange the data set into numerical 
order (see Figure 5). 


The revised example would be: 


MEDIAN 


AAs awe Aaeasa4s dD 





Figure 2.5. Defining the Term Median in the Example RAB Attribute Table. 

Next, find the middle of this ordered data set. If the data set contains an odd 
number of values, there will be one middle value; if the set is even, there will be two 
middle values, and the median is calculated by averaging these two middle values. In the: 
example, the data set contains a total of 7 values; therefore, the middle value for each 
attribute is listed in the 4°" column (see Figure 5). The problem with this data set is there 
is a 4-way tie among the attributes, and the data set provided does not give an ordinal 
ranking, therefore the decision-makers do not know which attributes are ranked 1%, 2" 
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and so on. The median method does not give the optimal solution; however, it explains 
where the RAB members have a general tendency towards an alternative, while not 
defining which alternative is optimal. 

Mode is the measurement that occurs most frequently in the data set. Referring to 
Figure 6, the mode 1s identified for each attribute. Attributes 1, 2, and 3 do not have a 
mode; therefore, the decision-maker cannot rank this data set to derive the piece 
choice among the RAB members. - 

When the decision-maker assigns equal importance among all the attributes, the 
simplest way to obtain an overall ranking 1s to total each row and rank them in ascending 
order (Yoon and Hwang, 1995). This technique is called the Nominal Group Technique, 
which provides an ordinal ranking of the alternatives; yet has its downfalls. First of all, it 
does not consider conflicting scores given by the RAB members. For example, in figure 
4, RAB(A) ranks Attribute 4 as the best (score of 1) and RAB(E) ranks Attnbute 4 as the 
worst (score of 7). This technique is a consensus building technique, where arithmetic 
operations do not provide the optimal solution, but presents a data set that every voting 
member can support. Figure 6 is a revised Delphi table listing the covered aggregation 
methods: 

2. Advanced Methods of Aggregation 

a. Median Ranking Method 
Median Ranking Method is used to measure the amount of agreement or 

disagreement among rankings. The method is done by a metric or distance function for 


the set of rankings. Median and consensus ranking is calculated by minimizing the total 
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absolute distance. Using the scores given in Figure 4, the method explained by Cook and 


Seiford (1978) follows. 


There are 7 RAB members and 7 attributes. The formula that is used to 
derive a distance matrix is represented by 
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Figure 2.6. Example RAB Attribute Table with Listings of Mean, Median, Mode and 
NGT. 


where dj, represents the distance matrix, k defines the number of rankings (or, in the 
distance matrix, the column number). In this research, each RAB represents one ranking. 
The symbol / represents the number of members (in this case, RAB identification). The 
symbol i represents the number of attributes (or, in the distance matrix, the row number). 
The distance matrix is the sum of the eichathiaas Saahaetain a selected ranking (k) and the 7 


given member rankings (a;’) of an attribute: 


34 





diy = [5-1] + [2-1] + [3-1] + [4-1] + [1-1] + [6-1] + [7-1] =21 
d73 = |6-3] + |7-3] + [2-3] + |1-3] + |5-3| + [4-3] + |6-3] = 16 


The complete calculation gives: 


DISTANCE MATRIX (dix) 





Figure 1.7. Distance Matrix of RAB Ranked Attnbutes Using Median Ranking Method. 

After calculating the distance matrix, the Hungarian method can be used to 
minimize the total distance. The output of the Hungarian method should give the optimal 
ranking. 

b. Hungarian Method 

The output of the Median Ranking method is the input to the Hungarian 
method. The Hungarian method is used in minimization problems in operations science 
such as minimizing completion time of a project, minimizing total cost, or minimizing 
the total distance among attributes. Referring to the distance matrix in figure 7, the 


following steps are taken: 


® For each row, subtract the minimum number in that row from al] numbers 
in that row. 
@ For each column, subtract the minimum number in that column from all 


numbers in that column. 


® Draw the minimum number of lines to cover all zeroes. If this number = 
m, stop. An assignment can be made. 
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® Determine the minimum uncovered number (call it @) 
a. Subtract d from uncovered numbers. 
b. Add d numbers covered by two lines. 
c. Numbers covered by one line remain the same. 
d. Then go to the previous step. 


After completing the above steps, the distance matrix now looks like: 





Figure 2.8. Minimization of the Distance Matrix. 


The steps to determine the optimal solution are: 
e Find a row or column with only one limited zero and circle it. 


® If the circle is in a row with one zero, draw a line through its column. If 
the circle is in a column with one Zero, draw a line through its row. One 
approach, when all rows and columns have two or more zeroes, is to draw 
a line through one with the most zeroes, breaking ties arbitrarily. 


® Repeat the previous until all the circles are lined. If this minimum number 
of lines equals m, where m is the number of jobs and workers, then the 
circles provide the optimal assignment 
(htpp://isgs.ba.ttu.edu/classes/jburns/Tsq5342/slides/ch07). 
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The final outcome of the assignment problem using the Hungarian method is: 





Score 


Atte [Rank [| Sear | 





Figure 2.9. Assignment Matrix. 

3. Assigning Weights to the Aggregated Ranking 

Once the final ranking is determined through the Hungarian method, the decision- 
maker needs to compute the weights for the attributes with respect to one another. In 
other words, how much more important is the highest-ranked attribute to the second 
highest-ranked attribute? The “Smarter” method, used in a software program called 
Logical Decisions® for Windows™ (LDW), uses the final ranking from the ageregation 
method as its input, and a least preferred and most preferred level is given by the 
decision-maker to compute the weights for each attribute accordingly. For example, 


there are three attributes, project time, completeness and total cost, ranked as first and a 
tie for second respectively (see Figure 10). 


Attribute Rank Least Preferred Most Preferred 
Level Level 


Figure 2.10. Example of Assessment Using the “Smarter” Method. 











3] 











The “Smarter” method will take the ranks and assign a higher weight to project 
time, and then, lower weights to completeness and total cost. For each choice sediment 
technology, there will be a level for each attribute listed in Figure 10. For example, 
sediment technology may take 3 years to be 80% complete, and a total cost of $7 million. 
The total utility score is calculated by the “Smarter” method with these figures and the 
rank of each attribute. After all utility scores are calculated for each sediment 


technology, the technology with the highest utility score is the optimal decision. 
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Ill. DESIGN IMPLEMENTATION AND RESULTS 


A. INTRODUCTION 

This chapter outlines the methodology used to implement Value Tree Analysis 
through the use of the Delphi Method conducted via the internet. The first four sections 
describe the research of applications used to create, manage and publish a website and 
survey, and use of a file transfer service to upload and download files to web servers. 
The fifth section discusses the formulation of roles of proxy-Restoration Advisory Board 
(RAB) members. The sixth section explains the implementation of the Delphi method 
via the internet-based discussion forum and survey. The process and results of the 
discussion phase and the two voting rounds are detailed. The last section provides the 
overall results of the survey experiment. 
B. WEBSITE DESIGN 

1. Requirements 

RABs need a common medium that provides access to sediment technology 
treatment information to educate themselves as well as others on important public issues, 
and to communicate their concerns and desires to public decision-makers. The 
instruction and the methodology information provided on the web provides another way 
to implement Delphi, which typically uses methods including conference meeting and 
mail out services in the first phase of execution. An electronic-based discussion forum 
provides proxy-RABs with the ability to post messages about their concerns, ask 
questions, read others’ messages, reply to any posted message, and chat if they wish to do 


so. An advantage using internet and web technology is to allow the proxy-Restoration 
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Advisory Board (RAB) members to meet virtually to discuss and to post their comments, 
questions and responses at their convenience. 

Z. Website Design Tools and Free Web Server Providers 

The authors of this research considered using several different website design 
tools to facilitate the Value Tree Analysis and the Delphi process. A brief discussion of 
the products considered is provided. Ultimately, the researchers selected Netscape® 
Composer, WS_FTP95, Xoom.com and Delphi.com for the research. Reasons for the 
choices are provided. 

Microsoft® FrontPage® 98 is a tool to create the web site. “Microsoft® 
FrontPage® 98 consists of the FrontPage® Explorer (a site management tool), the 
FrontPage® Editor (an HTML processor), image composer (with which you can create 
and edit both still and animated graphics), and three web server programs (from which 
you choose depending on your platforms)” (Tauber and Kienen, 1998). Microsoft® 


FrontPage® 98 has the following capabilities: 


e Creates and edits HTML easily, previews the page created and views the 
HTML source code 

® Designs tables, resizes rows and columns quickly 

® Includes forms where the data can be routed via e-mail or saved to a server 


e Includes Microsoft® Office documents in HTML pages 

e Includes text animation and other special effects 

® Assigns permission to view and change various parts of the designer’s site 
Netscape® Composer is another tool used to create or edit web pages containing 


images, tables and links; and publishes the final drafted website on the web. Netscape® 
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- Composer is easy to use, and its simplicity is similar to using a word processor. The files — 
that the composer creates are HTML (Hypertext Markup Language) files just as 
Microsoft® FrontPage® creates. When the web pages (HTML files) are published on 
the web, the browsers know how to interpret the HTML codes to display HTML files. 

Most of the Internet Server Providers (ISPs) include HTML editing tools to create 
a personal page. If the user is not a member of an ISP (such as America Online © 1999), 
there are many Free Web Service Providers (FWPs) available on the web to provide the 
necessary tools to create and host web sites for free. 

FWP services are good if the number of pages created do not exceed fifty. Most 
FWP providers have limitations on certain web page features, such as computer hard 
drive space (see Table 3.1). Therefore, if the user desires to have a separate domain 
name, create hundreds of pages, use different web technologies (such as secure 
waneaetions): and expect high traffic capacity; then FWP cannot be used. For further 
information regarding free web providers, refer to http://netguide.com. FWP providers 
usually provide services including free e-mail, file management and File Transfer 


Protocol (FTP). FWP providers usually generate revenues through advertising. 


Free Web Service Provider Hard Drive Space (MB) 


Xoom 1] 
Crosswinds Unlimited 


AcmeCity 20 


Angelfire > 


Hypermart 





Table 3.1. Examples of FWPs with Available Hard Drive Capacities. 
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Delphi Forums Inc., 1999, provides a free electronic medium for open discussion 
that allows members to create and manage personalized online forums. Delphi Forums 
provides a medium to connect seal lias ideas through a variety ee full-featured, web- 
based messaging services. Visitors have the capability to browse through messages left 
by others, respond to previous postings, or start new conversations. Delphi Forums 
includes real chat and file attachment features. In addition, the designer of the forum has 
the ability to limit the access by designating the forum as a private website vice a public 
website. For more information regarding the use of Delphi Forums, visit its website at 
www.delphi.com. 

This thesis project uses Delphi Forums because its services meet the requirements 
of this research. Delphi also provides tools to build a site homepage, which offers a link 
to the discussion forum. A limitation of Delphi Forums is the inability to allow more 
than one web page per website; therefore, another website provider is used to host links 
to the Delphi Forums homepage. The online forum is designed as a private website to 
allow only invited proxy-RAB members access to the forum. Proxy-RAB members are 
assigned unique usernames and passwords by the forum hosts. 

On the advice of fellow Information Technology students, Xoom.com, Inc. 1997 
is used for hosting the website created in this thesis research. The only requirement while 
registering with Xoom.com is to provide personal information including user name, e- 
mail address and affiliated company. To create the link pages, Netscape® Composer is 
used. After creating the desired web pages, the web designer publishes the web pages on 


a web server (Xoom.com). 
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This thesis research uses WS_FTP95 to upload the designed web pages (HTML 
pages) to the web server. To upload the web pages to the web server, the web designer 
must have access to the web server. During the uploading process, the FTP program asks 
the web designer to enter a profile name, host name, user name and password to be able 
to connect to the server and perform the necessary file transfers. 

3. File Transfer Protocol (FTP) 

“The most widely-deployed Internet file transfer service uses the FTP. FTP 
permits transfer of an arbitrary file, and includes a mechanism that allows files to have 
ownership and access restriction. A file transfer service can move a copy of a file from 
one computer to another, either interactively or in batch mode.” (Comer, 1997, p. 347, 
348) 

FTP is based on the client-server model of communication between a server and a 
client computer. The server provides services such as transferring the requested 
information to the client computers. The client computers request information from the 
server. Non-government and government institutions usually have an FTP server to post 
their materials to the interested parties or individuals. In this thesis, an application of 
FTP is used to upload the HTML apes that are created. 

C. SURVEY TOOLS 

This thesis research focuses on the subjective input of the stakeholders who are 
represented by proxy-RAB members. The importance of eliciting the concerns of 
members regarding the San Francisco Bay Sediment Project, and then allowing the group 
to rank the overall set of concerns ordinally, 1s required to utilize public opinion in the 
decision-making process. Currently, public representatives (RAB members) are asked to 
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attend local meetings, where discussion forums take place to identify the issues pertinent 
to the problem. The goals of utilizing a web survey are to eliminate the need for a 
physical meeting by conducting on-line discussions, and by eliciting issues over the 
internet. 

1. Research Requirements 

The survey tool must fulfill several functions. First the survey must provide an 
internet-based platform to perform the survey. Second, the tool must allow a large 
number of inputs by voting members. Third, the tool must allow members to rank the set 
of concerns (changed into attributes) in an ordinal manner. Fourth, the tool must provide 
the individual results of the survey to the survey host via email in text or database format, 
and finally, the tool must provide the entire service free of charge. 

a Alternative Survey Tools 

The internet provides : host of websites that allow the users to develop surveys 
free of charge. However, the free survey providers have limited capabilities in 
questionnaire design, storage and publishing results. The following free internet 
providers and its capabilities are explained. 

Delphi Forums, Inc. 1999 allows stakeholders to voice their opinions 
anonymously via ceaes discussion forum, the ability to rank a set of attributes 1s 
unavailable. Instead, Delphi Forums allows the forum host to list up to five choices, 
where the voting member may choose the most important choice alternative. In other 
words, the voting allowed in Delphi Forums allows the stakeholder to cast a vote in a 
ballot-type format. For more information regarding the use of Delphi Forums, visit the 


website at www.Delphi.com. 
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Perseus SurveySolutions™ for the Web is an application for creating and 
distributing surveys, as well as collecting, analyzing and reporting results. While some of 
the most common applications used by Perseus.com include customer and saiplaves 
satisfaction surveys, course evaluations, and opinion polls, Perseus SurveySolutions™ for 
the Web also creates templates for lead generation forms, technical support inquiry forms, 
web-feedback questionnaires, and order forms. One of the most impressive features of 
Bees SurveySolutions™ for the Web is its ability to produce instant survey results, 
making Perseus SurveySolutions™ for the Web ideal for publishing entertaining and 
informative polls on a personal web site. For more information regarding survey 
development with Perseus SurveySolutions™ for the Web, visit its website at 
www.Perseus.com. | 

This thesis research uses SurveySolutions™ for the Web, developed by Perseus 
Development Corporation. This tool uses an intuitive word-processing interface for 
survey design, and automates HTML translation for web-surveys and e-mail conversion 
for surveys that will be distributed by electronic mail. Survey responses are collected 
either on the server or on a workstation. Results can be stored in a variety of formats 
including standard ASCII TSV (tab separated values) files on the server and Mictosett 
access databases on your workstation. Perseus SurveySolutions™ for the Web also 
allows a free 30-day tral version, which allows this thesis to be conducted at no cost. 

D. FINAL WEBSITE AND SURVEY DESIGN 
Upon choosing the free applications for creating, managing and publishing a 


website and survey, the creation of the NPS Sediment Project Home Page was formed 
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(www.delphi.com/nps survey). The research homepage and the survey are shown in 


Figures 3.1 and 3.2. 





NPS Sediment Project Home Page 





Background of the Sediment Project: 


As a result of implementation of Base Realignment and Closure Commission (BRAC) 
decisions, the naval operations in the San Francisco Bay area has declined precipitously. 
Yet a legacy of the naval presence remains; portions of the Bay are contaminated from 
years of naval operations. The San Francisco Bay Sediment project is the U.S. Navy's 
program directed at redressing the contamination of sediment from the Bay area naval 
operations near: Mare Island, Hunters Point Shipyard, Alameda Point and Treasure 
Island. The project is administered by Engineering Facility Activity (EFA), West which 
is responsible for the project design as well as project execution. Two students at the 
Naval Postgraduate School have designed a communication process (that EFA, West can 
consider for implementation) that could help EFA, West elicit and utilize public concerns 
regarding site cleanup. The methodology is specifically aimed at assisting EFA, West 
officials in selecting the best remedy technology for each of the contaminated sites. The 
methodology, designed by the students, involves the public in the technology choice- 
decision making process and ensures that these concerns are a part of the decision- 
making for éach site. For more information, please go to the link entitled "Methodology 
Information." Proxy-RAB members, those designated as participants in the methodology 
experiment, are instructed to read the "Methodology Information" before reading the 
"Instruction for Proxy-RABs" link. 


Relevant links: 


Message Board 

Chat 

Treatment Perspectives 

NAVFAC Homepage 

Navy's 5-year environmental restoration pgm 
Sample Survey 

Methodology Information 

Instruction for Proxy-RABs 

Survey-Phase2 

Survey-Phasel Results 


Figure 3.1. NPS Sediment Project Home Page. 
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NPS Pilot Sediment Project Survey 


Please review and rank the following attributes according to your personal preference 

(1 = most important, 24 = least important). Do not assign any ties. Feel free to use the 'other' 
section to add any attribute you feel is important, but not included. 

Thank you for your participation. 


Attributes. 
# of domestic pet deaths 


# of domestic pet poisonings 

# of civil lawsuits filed 

# of complaints or protests 

# of on-site injuries 

Rate of medical treatments in area 

# of birth defects in area 

Amount of contaminant disposal 
Distance of disposal site from residential area . 
Contaminants in air 

Contaminants in soil 

Quality of drinking water 

Migration rate of drinking water 
Availability for recreational boating/fishing 
Total cost of site cleanup 

Amount of total depreciation of land 

# of fish/wildlife deaths 

# of fish/wildlife poisonings 
Consumable seafood? 

Project completion time 

Stability of foundation (after treatment) 
Time for site to naturally restore itself 
Future reuse for residential area? 


Future reuse for business district? 
2. Please provide any comments about your ranking. If you have an additional attribute 


that should be considered, list them in the below box and provide justification for its 
Inclusion. 


Figure 3.2. Survey used for Delphi Rounds I and II. 
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E. IMPLEMENTATION OF DELPHI METHOD 

The general steps of the Delphi method are outlined in the literature review 
chapter (Chapter 2, Section D.1.b). This section describes the actual methodology used 
in eliciting stakeholder value and aggregating the data to give a final ordinal ranking of 


the stakeholders’ set of attributes for selecting the optimal sediment technology 


treatment. 

1. Proxy-Restoration Advisory Board (RAB) Selection 

The thesis researchers originally planned to use actual RAB members involved in 
the San Francisco Bay Sediment Project; however, time constraints and other 
commitments by EFA, West members limited their participation at this time. The 
available information about sediment treatment perspectives and NAVFAC’s Five Year 
Restoration Program) is used to develop the proxy-RAB roles. 

Proxy-RAB members with personal computers and internet access were chosen. 
Of the fourteen candidates originally requested to participate in the experiment, seven 
agreed to take a role as one of the seven panelists representing RABs from: Alameda 
Naval Air Station, Concord Naval Air Station, Mare Island Naval Shipyard, Moffett Field 
Naval Aur Station, Treasure Island Naval Station, Treasure Island Naval Station Hunters 
Point Annex, Oakland Fleet and Industrial Supply Center. The panelists’ demographic 
data include: 

- 0-5 Navy Flight Officer, stationed in Monterey, CA 

- Q-5 Marine Corps Reservist, stationed in Quantico, VA 


- 0-4 Retired Limited Duty Officer, residing in San Diego, CA 
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- 0-3 Naval Advisor, stationed in Jacksonville, FL | 

- O-3 Judge Advocate General Officer, stationed in Bahrain 

- O-3 Fleet Support Officer, stationed in Monterey, CA 

- O-3 Civil Engineer Officer, station in Monterey, CA 

Each panelist was emailed information specific to the site they represented. The 
roles of each RAB member are shown in Appendix B. 

zs Discussion Phase of Delphi 

As explained 1n the Literature Review Chapter, the goal of the discussion phase in 
the Delphi is to identify the public issues relevant to the San Francisco Bay Sediment — 
Treatment Project (Chap. 2, Sect. D.1.a). The proxy-RAB panelists are given a period of 
one week to review their specific role, and to identify the issues which are considered 
important in selecting the best sediment treatment technology. The panelists are 
instructed to go to the NPS Sediment Project Home Page (www.delphi.com/nps_survey), 
where the panelists have access to background information and other relevant links 
regarding sediment restoration programs. Instructions for participating in the discussion 
phase of the Delphi are also given, and panelists are directed to a this electronic 
discussion forum to provide their concerns about the sediment treatment project. The 
leading question posted on the discussion forum is, “As a Restoration Advisory Board 
(RAB) member involved in the San Francisco Bay Sediment Project, what are the most 
important issues regarding sediment treatment? (Provide a minimum of 3 issues)” The 
unedited responses of the participating proxy-RAB panelists are shown in Appendix C. 


The results of the discussion phase are interpreted into measurable attributes, and 
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constructed into a value tree representing all the issues articulated by panelists relevant to 
selecting the best treatment technology. 

Of the seven proxy-RAB panelists, only five responded to the discussion 
question. The five participants gave a total of 27 issues, 24 of which are relevant to 
choosing the best treatment technology. Figures 3.3 is the constructed value tree and 


Figure 3.4 summarizes the attributes articulated by the proxy-RAB panelists. 
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Select Scdiment Tec 
Goal 













Cost of Cleanup 






Land Depreciation 
Measure 


Domestic Pct Heath # of pet deaths 
Goal Measure 


Wof pet posonings 


Enviromment Air Contamimants 
Goal Measure 


Amount of Dsposal 
Measure 











Dist. frm Disp Site 
Measure 


Soil Contam imants 
Measure 


Water Issucs Boating/Fis hing OK? 
Goal Measure 


Drimking: Water Migr 
Measure 


Drinking Water Qua 
Measure 


Firh/w ildlife Healt Woof fAw de aths 
Goal Measure 


w# of fFAv posronmeges 
Measure 


Consumahie fAv? 
Measure 


Futurc Reuse Residential Use 
Goal Measure 


Use by Businesses 
Measure 


Human Health # of birth dcfccts 
Goa) Measure 


# of med. treatment 
Measure 


Project Efficacy Stabic foundation? 
Goal Measure 


« 
- 









Tame for restoratio 


Measure 
Project Time Project Compl. Time 
Goal Measure 
Public Acccptance # com paintsprotes 
Goal Measure 


# kaw sults filed 
Measure 


Public Safcty # of onsite inj uric 
Goal Measure 


Figure 3.3. Value Tree Derived from Proxy-RAB. 
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ISSUES PRESENTED BY PROXY-RAB PANELISTS 


# of domestic pet deaths: How many pets have died as a result of exposure to the 
sediment treatment technology? 


# of domestic pet poisonings: How many local pets can be infected with a disease if 
exposed to the sediment treatment technology? 


# of civil lawsuits: How many class action lawsuits have been filed as a result of 
exposure to the combination of contaminants and sediment treatment technology? 


# of complaints or protests: How many complaints have been filed or protests held in 
defiance to the sediment treatment technology? 


# of on-site injuries: How many people are at risk of being physically injured during the 
sediment treatment project? 


Rate of medical treatments: How many people will seek medical treatment for ailments 
such as cancer, respiratory problems or digestive problems as a result of the sediment 
treatment technology? 


# of birth defects: How many newborn babies will be infected with a disease as a result 
of the sediment treatment technology? 


Amount of contaminant disposal: What is an acceptable amount of contaminants that 
can exist without causing any ill effects on the San Francisco Bay? 


Distance of disposal site from residential area: How far does the disposal site have to 
be to not affect the San Francisco Bay area? 


Contaminants in air: What is EPA’s acceptable level of contaminants in the air? 
Contaminants in soil: What is EPA’s acceptable level of contaminants in the soil? 
Quality of drinking water: After treatment, will the water be potable or non-potable? 


Migration rate of drinking water: Since there is a possibility of contaminant runoff in 
the water, how much of the natural drinking water will be affected by the sediment 
treatment technology? 


Availability for recreational boating/fishing: Will the waters of the San Francisco Bay 
be safe enough to permit recreational boating and fishing? 


Total cost of site cleanup: What is the total cost for the sediment treatment technology? 
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Amount of total depreciation: After the sediment treatment technology has been 
applied, how much will the land depreciate in its market value? 


# of fish/wildlife deaths: How many fish/wildlife have died as a result of exposure to the 
sediment treatment technology? 


# of fish/wildlife poisonings: How many local fish/wildlife can be infected with a 
disease if exposed to the sediment treatment technology? 


Is seafood consumable? Will the seafood in the San Francisco Bay be consumable? 


Project completion time: How long will it take for complete restoration of the 
sediment? 


Stability of foundation after treatment: Has the contamination and proposed treatment 
affected the stability of the soil to support the foundation of future building in the area? 


Time for site to naturally restore itself: How much time will it take for the sediment to 
restore itself after the treatment technology has been applied? 


Future reuse for residential area: After completion of the restoration, will the treated 
areas be suitable for building a residential community? 


Future reuse for businesses: After completion of the restoration, will the treated areas 
be suitable for building businesses? 





Figure 3.4. Attributes articulated by proxy-RAB members for the Restoration Project. 
3. Delphi — Round I 
At the conclusion of the discussion phase, where proxy-RAB panelists articulate 
their relevant values (i.e. attributes), proxy-RAB panelists are asked to participate in the 
Survey-Phase I. The panelists are instructed to rank each attribute in an ordinal manner 


(1 = most important, 24 = least important), with no ties allowed. The panelists are given 
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three days to rank the attributes. Panelists are encouraged to refer back to the message 
board to ieiiciaiai each other’s opinion better before ranking.” 

Of the seven proxy-RAB panelists, only four participated in Round 1 of the 
Delphi. The results are summarized on a spreadsheet, where each attribute’s mean and 
standard deviation are calculated. The original plan to use the Median Ranking and 
Hungarian Methods to aggregate the ordinal rankings proved to be unsuccessful. A 
requirement for using the Hungarian Method is to have a square matrix, consisting of m 
rows and m columns. To achieve a square matnx, the original distance matrix (given by 
the Median Ranking Method) should be supplemented with dummy zeros. The small 
number of responses (four) versus the large number of attributes (24) calls for 20 
columns of zero values. After applying the steps of the Hungarian Method to the Median 
Ranking Method, only four attributes are distinguished as being more important than the 
remaining 20 attributes; and of those four attributes, no specific ranking could be: 
determined. The remaining 20 attributes could be ranked in an arbitrary manner, which 
does not provide a desired overall ranking for the attributes. Therefore, after collecting 
the input of each proxy-RAB member, the mean value is edicutated for the rank given to 
each attribute, and a standard deviation is calculated for each attribute to determine the 
distance from the respective mean. The results of the first round of Delphi are listed in 


Table 3.2. 


> On the third day of Round 1, the host received a message that the survey server would not accept the 
inputs of the proxy-RAB panelists. The survey was reformatted for the web, and re-published within hours. 
Due to this problem, an extra day 1s given for proxy-RAB members to rank the attributes. 
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4. Delphi — Round II 

In this round, the final ordinal ranking and a summary of Delphi Round 1 is 
published. One proxy-RAB panelist did not provide justification for his or her ranking in 
Round 1, and was asked to provide comments on the electronic message board. Panelists 
are encouraged to exchange information on the message board to educate each other, and 
at times defend their rationale for ranking the attributes. 

Next, proxy-RAB panelists are invited to participate in Delphi Round II, and are 
given two days to re-evaluate their personal ranking with that of the overall group 
ranking, and provide a second ranking with justification. Of the seven proxy-RAB 
panelists, five responded.* The results of Delphi Round II are aggregated by determining 
the mean and standard deviation of each attribute. The final ordinal ranking of attributes 
are provided in Table 3.3. 

F, RESULTS SUMMARY 

The objective of this section is to determine the degree of association between 
Delphi Rounds I and II. Determining the degree of association between the rounds is 
important because it confirms whether a group is converging towards an overall ranking 
of the set of attributes. 

In this thesis project, the mean value was calculated for each round because of the 
difference in the number of participants in each round of Delphi (four panelists in the first 
round, and five in the second round). This survey could not utilize the Nominal Group 
Technique (NGT) because the total for each attribute would increase in the second round 


4 
The Perseus server was down for two hours on the second day, and results could not be collected as 


scheduled; therefore, extra time was given for input. 


55 


by a value (given by the fifth panelist), thereby providing no level of convergence among 
the attributes. 

To determine the level of convergence of the rankings in Delphi Rounds I and II, 
the coretation of the oan values and standard deviations are taken. Since the 
calculation of the correlation provides a number close to one, it can be summarized that 
the mean ranking for each attribute in Delphi Round II are closely associated to the 
results obtained in Delphi Round I. In addition to the overall correlation obtained in the 
average rankings for each attribute in Delphi Rounds I and II, the average standard 
deviation associated with the rankings of the attributes declined in Delphi Round II, 
indicating more agreement among the ranks given by the proxy-RAB panelists. As a 
crosscheck to measure the degree of convergence between Delphi Rounds I and II, a rank 
correlation test 1s applied to the rank data, to find Spearman’s rank correlation coefficient 
(R). The result of 0.94 indicates correlation in Delphi Rounds I and II, where Delphi 
Round I has a positive influence on the ranking obtained in Delphi Round II. Because of 
the influence, this thesis can conclude that after two rounds of Delphi, the proxy-RAB 
panelists grew closer to consensus, and that further rounds would continue to produce 
high correlation coefficients close to one. Table 3.4 provides the overall correlation of 
the two Delphi rounds, and lists the convergence (if any) for the ranking of each attribute. 
G. ASSIGNING WEIGHTS TO THE ATTRIBUTES 

In Appendix A, different methods are explained to assign weights to the 
attributes. As a result of the implementation of the Delphi and aggregation methods, the 
final ordinal ranking of attributes are obtained. The rank sum weighting or rank 


reciprocal weighting methods are used to assign weights to the attributes. Both 
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techniques are approximations where the decision-maker may choose his or her method 
of preference. Furthermore, the researchers made the assumption that the degree of 
importance between each attribute is equal, as they are ranked ordinally. The 
assumption, although not one hundred percent theoretically defendable, was made to 
simplify the process. The results are thought not to be affected, while the assumption 
makes the process easier to legitimate to the stakeholders. The weights of the attributes 


are calculated and listed in Table 3.5. 
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IV. CONCLUSION AND RECOMMENDATIONS 


Based on the results achieved in the implementation phase of this thesis, the 
following lessons learned are identified. The lessons concern the participation level of 
proxy-RAB members, the articulated attributes alice and the effectiveness of the 
internet. 

First, any implementation of the internet-based Delphi should not expect 100% 
participation. This thesis research originally planned to use seven panelists to participate 
in its survey; however only four and five panelists participated in the first and second 
rounds respectively. The anonymity feature of the Delphi method may actually restrict 
the participants’ accountability in contributing to the discussions and ranking the 
attributes. In a typical Delphi, participants gather in a conference room and submit their 
votes. While the votes are anonymous, there 1s an assumption that each member submits 
a vote, since the effort to convene at a physical location makes the act of voting a small 
marginal effort. Due to the expected decrease in the participation rate over the internet, a 
recommendation may be to invite twice the number of panelists to obtain the desired 
input level. 

Second, since proxy- Restoration Advisory Board (RAB) members are used in the 
survey, the results might not reflect an accurate selection of issues related to the San 
Francisco Bay Sediment Project. Remediation technology selection is a technical subject 
and, with the exception of one panelist, the proxy-RAB members do not have the desired 
knowledge to participate this type of survey for reasons such as geographic location, real 


sediment issues in the San Francisco Bay area, and the actual objectives of existing 
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RABs. Better survey results can be obtained by using real stakeholders, and having 
technical experts available to answer any treatment-specific questions. 

Third, the flexibility introduced into the process by using the internet to facilitate 
the Delphi has positive and negative implications. The use of the internet allows for 
broader participation since participants are not constrained by geographic locality or time. 
However, this thesis research has found the internet-based Delphi method to require more 
time than the traditional Delphi method. Because participants rarely participate 
concurrently, panelists are required to visit the forum more than once to read the 
responses of others and to provide any additional comments. 

Fourth, the level of technicality may also influence the time it takes to complete 
the Delphi method. Highly complex issues may be easier to explain with traditional 
methods such as physical proximity. If the subject is technical, more time may be taken 
to educate and elicit values concerning the discussion topic. When the panelists meet 
physically, the exchange of information occurs much faster than by posting messages on 
an electronic bulletin board, and the technical subjects can be explained better by using 
visual aids to make a point. Also, there may be some hesitancy by some to utilize the 
internet. Techno-phobia may limit the usefulness of the internet. 

Online chatting is a viable option to use during the discussion phase of the Delphi 
method. Although proxy-RAB members have the ability to use the chat feature, the 
feature was not utilized by the members in this thesis research. This may be for a variety 
of reasons including time constraints and number of RAB participants. The time of day 
that proxy-RAB members choose to participate may not be the optimal time for others. 


This thesis research includes panelists from three different time zones (one being 
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overseas); therefore, the likelihood of the panelists being online at the same time is 
difficult. 

The Delphi survey results show that the Delphi established consensus through a 
degree of association. The ordinal ranking of round two is closely associated with the 
ordinal ranking of round one and the standard deviation of many of the attributes at round 
two decreased relative to round one. If one or more additional rounds of Delphi are 
conducted, more consensus might have been obtained. 

During the creating of the survey and gathering data, several difficulties were 
experienced. One of the difficulties was formatting the questionnaire. Due to the wrong 
formatting, panelists were unable to submit their responses for two days. Only after 
receiving a warning message from Perseus Corporation were the authors able to identify 
the problem and solve it. The survey server unexpectedly ceased operations for several 
hours, which the researchers have learned is fairly common with the use of free internet 
tools. Given the limited experiences from this research, the authors concluded that the 
services of Free Web Service providers might not be reliable enough to meet the 
requirements of organizations. The use of free internet tools and servers should be 
considered carefully. Non-government and government institutions should have control 
of the server and not depend on others for full-time uninterrupted service. 

At the end of Delphi Round I, proxy-RAB panelists identified public awareness as 
an issue that should be considered in the San Francisco Bay Sediment project. Since this 
does not relate directly to the choosing the optimal treatment technology, it is not 
included in the value tree. There is concern by the panelists, however, as to how the 
public will be informed about the issues, results, remediation technologies, and updates 
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on the sediment treatment projects. This point re-confirms the importance of this thesis 
research and its goal of including public involvement and awareness, and of gathering 
public opinion into a value tree that can be used by decision-makers to make the optimal 
decision. It also confirms that the public does have a concern with the decisions made, 
and with the gathering of public opinion, decision-makers are better prepared to 
legitimate their choice to the public. | 

The original plan of using the Median Ranking and the Hungarian aenea 
changed during the implementation phase. After numerous attempts of applying the 
Hungarian method to the data, the researchers found that it is impossible to use a matnx 
that is made up primarily of zeros. The Hungarian method uses the distance matrix 
derived in the Median Ranking method, and because the Hungarian method requires a 
square matrix or a matrix close to a square matrix, an assignment could not be derived 
from the distance matrix calculated in this research.” Therefore, the basic methods of 
aggregation and statistical methods including standard deviation and correlation 
coefficients are used. 

The use of the internet is a viable method in eliciting public values; however, 
there are issues that should be considered. If an organization wants to gather public 
opinion from people living in a certain area, the organization may have problems in 
access control to only allow local residents to participate. Another issue is if the number 
of people involved is relatively high, then the results should be stored in the database. 
This requires web database interaction, and presents a new opportunity for thesis work. 


> See section E.3 of the Implementation and Results Chapter. 
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APPENDIX A. ADDITIVE UTILITY FUNCTION AND 
MULTIATTRIBUTE UTILITY THEORY (MAUT) 


A. VALUE AND CHOICE 

In his book, “Making Hard Decisions,” (1996) Clemen argues, “the basic reason 
for using a utility function as a preference model in decision-making is to capture our 
attitudes about risk and return.” (p. 530) Risk and return are considered elementary 
objectives in decision-making, and usually conflict with one another when pursuing high 
returns while minimizing risk. Clemen (1996) provides an example of a mayor deciding 
whether to approve or disapprove a new electric power generating station. The known 
factors in this situation include the fact that the city needs more power capacity; but, 
building the new plant would worsen the city’s air quality. The mayor might consider: 
health of residents, economic conditions of the residents, psychological state of the 
residents, economy of the city and the state, businesses, and local politics. 

A preference model can assist the decision-maker (in this case, the mayor) to 
select a single alternative from a set of feasible alternatives. Consensus building could be 
used, which provides a simple way of prioritizing the issues in a manner, which the 
decision-makers can support; however, more mathematical methods of aggregation may 
be preferred. 

1. Objectives 

Decision analysis has been used successfully to construct and solve problems for 
numerous situations. The analysis is based upon choosing a solution approach that lists 


the decision-maker’s objectives, and assigns a weight according to its importance. When 
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there are two or more conflicting objectives, an additive preference model may be used to 
calculate a utility score associated with each objective. These objectives are weighted 
appropriately, according to the relative importance to each other, and then added together 
to give an overall solution to the problem. 

a. Fundamental versus Means Objectives 

To develop a structured decision-making process, objectives for solving 
the problem must be defined. Objectives can be thought of as a “desired end-state 
resulting from planning activity,” (Klemdorfer, Kunreuther and Schoemaker, 1993, p. 36) 
and can be labeled as either fundamental objectives or means objectives. 

Fundamental Objectives are objectives that the decision-maker wants to 
accomplish. Means Objectives are objectives that help accomplish the fundamental 
objectives by defining measured attributes. For example, “working fewer hours may 
appear to be an important objective, but 1t may be important only because it would allow 
an individual to spend more time with his or her family or to pursue other activities that 
represent fundamental interests, things that are important simply because they are 
important. Thus, minimize hours worked is a means objective, whereas maximize time 
with family is a fundamental objective.” (Clemen, 1996, p. 44) 

Attributes are important in explaining and quantifying the higher level 
values and concerns, and are defined as the criteria for measuring objectives. According 
to Rowe (1992), an attribute is the quantity measured on an attribute scale. Attribute 
scales are used as a tool to measure the accomplishment level of the fundamental 
objectives (Clemen, 1996). For example, in buying a car, minimizing cost may be an 


objective, and measuring cost 1n dollars is an appropriate attribute scale. 
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b. Objective Requirements 


Clemen (1996) states that several conditions must exist for objectives and 


attributes to be effective tools mn decision-making: 


The set of objectives (as represented by fundamental objective hierarchy) 
should be complete. All relevant aspects of a decision should be included. 
If the decision-maker is not satisfied with the result or is unwilling to 
accept the results, then the set of objectives would have to be considered 
incomplete. 


The number (or set) of objectives should be as small as possible. Too 
many objectives can make the decision much harder to address. In this 
case, hierarchy is significant because each objective provides a useful 
representation of objectives that are important to the decision-maker, and 
each objective is different from the available alternatives. 


The set of objectives should not be redundant meaning objectives should 
not be repeated in a hierarchy. If there are redundant objectives, the 
optimal alternative may not be chosen. 


The set of objectives should be decomposable. The decision-maker 
should be able to think about each objective separately without having to 
consider others. For example, a student finding an apartment to rent may 
look at attributes such as monthly rent and distance from school. The 
student can think about these attributes separately: regardless of rent, it 
would be preferable to live closer to school; or, regardless of distance, it 
would be preferable to have low rent. Therefore, the objectives would be 
decomposable into the two attributes of cost and distance, which can be 
considered independently. If the attributes are related in a way that does 
not permit decomposition, then the set of objectives has to be changed. 


Means and fundamental objectives should be distinguishable. As defined 
above, means objectives accomplish fundamental objectives through 
attributes. If fundamental objectives are not well defined, there is no clear 
reason why a decision must take place. 


Attribute scales must be operational. These scales must provide an easy 
way to measure performance of each alternative or the outcomes of the 
fundamental objectives. The attribute scale of cost in dollars for buying 
an economical vehicle is a good example of this rule. 
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c. Conflicting Objectives 

When two or more objectives conflict, the challenge of recognizing the 
differences between each objective, and choosing the method that comes closest to 
solving the problem is difficult. Through implementing trade-off analysis, a decision- 
maker can add subjectivity by deciding how best to trade-off a more important (or 
dominant) objective for a lesser one. 

When only two objectives conflict, the trade-off analysis is not difficult. 
Each alternative could be listed or ranked according to how well it meets each objective. 
By plotting on a standard X-Y graph, the decision-maker can move systematically in one 
direction, through the alternatives, switching one alternative with the next. The benefit to 
this method is that once a switch is made, the preferred alternative must be better, and 
there is no need to go back and compare to a lesser-preferred alternative. 

2. ADDITIVE UTILITY FUNCTION 
An additive utility model is a way of evaluating conflicting objectives by 

calculating an overall utility score. The overall utility is the "weighted sum of individual 
utility functions for each fundamental objective.” (Clemen, 1996, p. 561) For the 
purposes of this research, the additive utility model can be used to choose the best 
remediation technology. The Additive Utility Model produces an ordinal ranking of the 
alternatives. Assume that there are individual utility functions U)(x)),.....,.Um(xm) for - 
different attributes x, through x,,, and that each function assigns a weight of 0 to the least 
important and 1 to the most important attribute for that particular objective. The additive 
utility function is defined as a weighted average of these utility functions. (Clemen, 1996) 


Therefore, for m different attributes (x; through x,,), the utility of this outcome is 
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UQ1,.. Xm) = kU (x1) +... + KmUm(Xm) , or 


m 
= XkiUi (x) 

where the weights (k;,...,Km) are positive and add up ‘to 1. 

a. Utility Curves 

Utility curves provide a graphical representation of the utility score in 
response to a change in attribute measurement. Figure A-1 depicts three types of utility - 
function curves that represent a translation of an attribute into utility units. The curves 
are dependent upon expert evaluators, who through subjective input, determine factors 
such as the attributes to be used, the range of scores for each attribute, and the conversion 
factor for a score to change into a utility score. The downward curve in figure A-1(1) 
represents a decreasing utility function for an attribute, meaning, as the attribute increases 
towards the higher level of its defined range, the utility score of that attribute has 
decreased. For example, in the sediment treatment project, the sooner the project is 
completed, the better the satisfaction level of the objective. Therefore, time 1s the defined 
attribute, and the experts of the project have defined minimum and maximum total 
project time for each remediation technology to complete the task. If the project is 
completed in minimal time, the utility assessed is high (closer to 1); however, if the 
project takes too much time and is measured towards the upper range level, a low utility 
is assessed, making that particular sediment technology less desirable considering. the 


time attribute only. 
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Figure A-1. Examples of Utility Function Curves. 


The second curve (Figure A-1(2)), represents a higher-scoring utility for a 
low measure of the attribute. However, as the level of the attribute increases, the utility 
soi decreases sharply until a certain level, and after a point the utility score is constant. 
For example, if the decision-makers want to complete the sediment project at a low cost, 
a higher utility will be given to those alternatives whose cost falls within the lower range. 
The third curve (Figure A-1(3)), depicts a linear curve, displays a direct relationship 
between an increasing attribute level and an increasing utility score. An example of this 
situation would be the desired level of completeness (progression of cleanup) for the 
sediment project, where the attribute moves towards a utility score of one, as the project 
becomes more complete. 

b. Indifference Curves 

Indifference curves graphically demonstrate the set of alternatives 


among which a decision-maker is indifferent. In sketching indifference curves, the 
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hihi will find that each slope produced from the alternatives is related to the 
trade-off rate that was assessed. For example, a decision-maker may be indifferent 
between two sediment treatment technologies, when considering the attributes of cost and 
time. The total utility is the same for both alternatives, therefore, the decision-maker is 
indifferent between choosing the two alternatives with respect to cost and time only. The 
slope could also be defined as the marginal rate of substitution, or the rate at which one 


attribute can be used to replace another (Clemen, 1996) (see Figure A-2). 


High is 


PROJECT 
TIME B 


Low 


Low High 
PROJECT COST 


Figure A-2. Indifference Curve. 
3. ASSESSING INDIVIDUAL UTILITY FUNCTIONS 
There are different methods regarding the assessment of the individual utility 
functions including the assessment of ratios, the calculation of proportional scores, and 
the standard utility function assessment. In this appendix, the simpler methods (ratios 
and sroporiiodal scoring) are explained to offer a choice among determining utilities (or 


utility scores). For additional information, refer to Clemen (1996). 
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a. Ratios | 

Ratios present a way of determining utilities based on a numerical 
comparison of attributes, whether the attributes are quantitative or qualitative. This 
method converts performance scores into utility scores, where the range of the utility 
scores are from zero (lowest level utility) to one (highest level utility) (see Figure A-1). 
Clemen (1996) uses the following example: the decision-maker decides that color is an 
important attribute in purchasing an automobile. Using the ratio approach for qualitative 
measures, the decision-maker determines a level of importance in comparing one 
attribute to another. In this example, the decision-maker decides that blue is twice as 
good as red, and that yellow is 2 % times as good as red. Points ranging from 0 to 100 
can be given to each alternative based on its performance on the attnbute. In the 
example, if 30 points were assigned to red, then 60 and 75 points would be assigned to 
blue and yellow respectively. These results can be calculated to reflect the conversion of 
the highest and lowest performance scores to the utility scores of 1 and 0 respectfully. 
First, constraints a and b need to be determined so that: 


0=a+b(30) 
1 =a+b(75) 


Solving these two equations simultaneously gives 


= - 2/3 
b = 1/45 


Applying these scaling constants, the utility values for the car color are: 
U, (red car) = -2/3 + (1/45)(30) = 0 


U, (blue car) = -2/3 + (1/45)(60) = 2/3 
U, (yellow car) = -2/3 + (1/45)(75) = 1 
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The utility of the car color could be used in conjunction with trade-offs determined for 
attributes such as price and fuel economy. 

b. Proportional Scores 

Proportional scoring is another method that can be used to assess utility 
scores. This method assumes natural numerical measures exist (i.e. dollars and years). 
The lowest and highest performance measures on the attribute are assigned utility scores 
of 0 and 1 respectively. The calculation in this method determines the utility scores for 
any intermediate performance measure that exists between the lowest and highest 
performance measures. 


The general formula is used: 


U;(x) = __x-— Lowest Performance Score 
Highest Performance Score — Lowest Performance Score 


xX — Xj 
Xj — X) 
Both the ratio and proportional scoring methods provide a way to calculate 
the utility scores for performance measures. However, another need in the additive utility 
function 1s to determine the weights of each attribute. Some of the available methods for 
assessing weights are explained in the next section. 
4. ASSESSING WEIGHTS 
Weighing the attributes is necessary to execute the Additive Utility Function. 
Weights reflect the importance of each attribute relative to the other attributes. There are 
various methods to assess weights including equal weighting, weights from ranks, ratio 
weighting, the pricing-out method and swing weighting. The simpler methods (equal 


weighting and weights from ranks) are explained in this appendix. In this research, the 
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“Smarter” method in Logical Decisions for Windows™ (1997) software is used to 
calculate weights in a srmple manner. This computer-based application allows the user to 
calculate weights more expediently than by hand, and providing a user-friendly platform 
that automatically provides an overall assessment of evaluation to the decision-maker. 
For further information on the other available methods, refer to the writings of Clemen 
(1996) and Edwards and Newman (1982). 

a. Equal Weighting 

Equal weighting is the easiest method to weight the attributes. The decision- 
maker simply assigns equal weights to each attribute. Thus, every attribute has the same 
weight. Equal weighting is sometimes useful when the decision-maker does not have 
meaningful weights obtained from the stakeholders or does not have weights at all. 
(Edwards and Newman, 1982) In this thesis research, the level of importance of each 
attribute is not the same, and spnsiderel important in selecting the optimal sediment 
remediation technology. 

b. Weights From Ranks 

In this method, the decision-maker has to have an ordinal ranking of attributes 
(listing the most important attribute first, the second most important attribute second, and 
so on). A numerical weight is then assigned to each attribute according to its importance. 
These methods provide a simple way of assessing weights; therefore, the decision-makers 
can utilize these methods to legitimate their decision better to the stakeholders. There are 
two methods of doing assessing weights from ordinal ranking: 


e Rank Sum Weighting: Assign the largest number to the most important 
attribute, the second largest number to the second important attribute, and 
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so on, until the list important attributes receives the rank of one. Then 
dividing each numerical number by the sum of all number gives weights. 


e Rank Reciprocal Weighting: Assign the numerical value of one to the most 
important attribute, two to the next most important attribute, and so on. 
The reciprocal of each numerical number is normalized. The normalized 
numbers are used as the weights. (Edwards and Newman, 1982) 
C. CONCLUSION 
The objective of using multiattribute decision-analysis techniques is to construct a 
model that 1s reasonable representation of a decision-maker’s value structure. The 
Additive Utility Function provides a weighted sum of individual utility functions for each 
fundamental objective. There are various methods available to calculate individual utility 
scores and weights where the decision-maker may choose the method of preference. Asa 


result, the additive utility model enables the decision-maker to select an alternative from 


a set of feasible alternatives. 


Ti 
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APPENDIX B. ROLES FOR PROXY-RESTORATION ADVISORY 
BOARD (RAB) MEMBERS 


The following is an example of the roles assigned to each proxy-Restoration 
Advisory Board (RAB) member. There are a total of seven roles assigned, with one 


participant assigned to each role. 


In your role, you are a concerned public citizen (member of the RAB — Restoration 
Advisory Board). Think about issues or concerns you may have regarding this site 
restoration. Your input will be used to choose the proper technology to remediate the 
polluted sites in the San Francisco Bay area. 


CONCORD NAVAL WEAPONS STATION (NWS) 


Background 
1. Demographics- Concord Naval Weapons Station (NWS) is about 35 miles 


northwest of San Francisco. It is surrounded by the city of Concord to the west 

and south (population 116,000); the city of Bay Point to the east (population 

17,000); and the small town of Clyde (population 600) to the north. The size of 

NWS is 13,023 acres. It is the major Naval munitions facility on the west and, as 

an ocean terminal facility, is used to transfer ordinance from trucks and rail cars 

to ships and vice versa. 
2. Contaminants — Concord NWS is contaminated with heavy metals, Petroleum, 

Oil, and Lubricants (POLs), volatile and semi-volatile organic compounds. 

3. Environmental Risk — 

a. Hydrogeology - Concord NWS is bound on the north by Suisun Bay and on 
the south and west by the city of Concord. Surface water and sediment is the 
pathway of greatest concern due to the interconnection of the Tidal and 
Litigation areas. Groundwater at Concord NWS is not used for drinking 
water due to its high Total Dissolved Solids content. 

b. Natural Resources — Suisun Bay is a transition zone between saltwater and 
freshwater ecosystems and is interconnected to the Concord NWS wetland 
areas. This area contains a diverse population of fish and other aquatic 


wildlife. The Bay is also used for recreation. The upland and the wetland 
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federal and state designated threatened and endangered specious. These 
include the Salt Marsh Harsh Mouse, California Clapper Rail, California 
Black Rail, Tule Elk, and the figwort family of plants including the Delta 
Tule Pea, and Soft Bird’s Peak. 

c. Risk— Four sites are ranked as high relative risk in the DOD Relative Risk 
Site Evaluation System at Concord NWS primarily because of the threatened 
and endangered species in the sensitive wetland areas and recreational users 
in adjoining Suisun Bay. The close proximity of NWS to the Contra Costa 
County Water Wells surrounding Mallard reservoir has also contributed to 
the high relative risk ranking. Risk to human health and the environment 


have been reduced due to the remedial action for the Litigation Area Sites. 


Public Involvement 
1. Restoration Advisory Boards (RABs) — There are 10 active RAB members. 


The Navy and the regulatory agencies have given technical presentations 
during the monthly RAB meetings. 
2. Relevant Issues — An Information Repository was established at the Central at 


the Central Contra Costa Public Library. 


After reviewing this page, please go to the Delphi Website at 


http://www.delphi.com/nps_ survey. There you will read the background information on the 
San Francisco Bay Sediment project, and asked to go to the Methodology Information and 


Instructions \inks. 


* Information taken from DON Environmental Restoration for FY 1998-2002. 
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APPENDIX C. ARTICULATED RESPONSES FROM PROXY- 
RESTORATION ADVISORY BOARD (RAB) MEMBERS 


The following responses are the results of the discussion phase of the Delphi 
method. The seven-day discussion phase resulted in five responses by proxy-RAB 
panelists. The results were then interpreted into measurable attributes relating to the San 
Francisco Bay Sediment Project, and placed in a value tree (see Figure 3.3 of in 


Implementation and Results Chapter). 


om: MERSING,OOSOC~—SCSCCTTCCCCC*‘( 
oan OOCOCOSOOCOCC“‘#TSNNC 
116.1 


HAs a citizen of this area I am concerned with three major factors HUMAN, 
ENVIRONMENTAL and COST. 
if AN ISSUES: 
| What provisions will be taken to insure “Public Safety’? 
i] he Government will cover —What health issues? (Birth defects, Cancer, 
Respiratory, Digestive) 
{+ Will public notices be given? (Newspaper, Radio, TV) 
{- Will the local community be allowed to take part in the planning of the 
lirestoration? 
j- What effects will the area have on our pets? 
What is the level of contaminants in the water supply? 
ENVIRONMENTAL: 
i-How long will the clean up take? 
-What is the schedule? (Start, Completion) 
How many phases are planed for the clean up and what are they? 
- Will the public be given a map of the contaminant area? 
|: What environmental studies have been conducted and will they be shared 
ith the public? 
j+ What is the acceptable levels of contaminants in the air, soil and water? Do 
hey meet EPA standers? | 
-How long will it take for the earth too naturally restore the area? 
j- What harmful gases will be in the air and soil? 
i: What is the short and long-term uses for the area? 
-What effects has the contaminants had on the stability of the soil to support 
{the foundation of future building in the area? 
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{/ What will it cost to conduct the clean up? 
|/- Who will absorb the cost? 
|- Will this lead to a deprecation of the local area? 
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importance, because it will determine the amount of clean-up that | 
iis required. | 


iicontamination of the bay? - This is the biggest environmental risk, : 


bay), and it doesn't directly impact development, 1t may be a low 
lipriority to the public. However, the Navy needs to plan for the 





From: FETHIYE1 | | 
| 0: MEMOCAN unread _ | 


5.3 inreply to 5.1 





at will the facility be used for in the future? - This 1s of primary 














Can we transfer the land prior to clean-up? - This allows the Navy | 
0 stop paying for operating costs (security, fire, etc.) during the 
lean-up effort. However, private landowners are very concerned 
ith their liability upon transfer. 

at remediation technology is proposed to prevent 








yet because it is a long way off (slow flow of pollutants to the 






remediation effort--whether it be in situ or via extraction (e.g., 
b1oslurper technology). 
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5.2 in reply to 5.1 


ere will the sediment go once it 1s treated? 
Realizing pollution of San Francisco Bay 1s possible what percentage chance 


ould you assign that collateral pollution may be caused by the cleanup 


effort? Collateral meaning airborne and waterborne pollutants cone 


ithe bay/air in the surrounding area as the cleanup progresses. 
at measures will be taken to ensure the safety and health of the 


surrounding population? 
at impact will the cleanup have on the surrounding wildlife and fisheries? 


How long will the cleanup take? 


Edited 11/24/99 6:13:32 PM ET by PWHITE27 
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I am a concerned public citizen and member of the RAB — Restoration Advisory Board. I 
Ihave three major concerns: 
111) How can I be assured that the Navy 1s accurately informing the public of the hazardous 


jimaterials stored (past and present) at Concord NWS. 

) If the groundwater at Concord NWS is not suitable for human consumption, how can I 
be assured that water fowl are not at risk of chemical exposure. What measures have been 
aken to ensure that water fowl safety? 

113) Likewise, are local residents that use Suisun Bay for recreational boating and fishing 
jjsafe. Is it safe to consume fish and wildlife from Suisun Bay (but not safe to drink the 
ater)? 
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[[From: SERENEDAY _ __ Jann 
||To: ALL CI of 1) 


i1As a member of RAB, there are 3 main concerns: 
{/1. Short/long term effects of exposure to both known and unknown contaminates to ila : 
112. Safety of wildlife. : 
(3. Migration of the contaminated water to populated areas. 
Points: 
{Concerns 1 and 3: This site is a known place for migratory fish, shellfish, birds, and . 
| mammals, including, crab and shrimp. Fish from this area can, and probably will migrate to’ 
| populated fishing areas. There has been no study to show the effects of human consumption 
ltof such contaminated foods. Additionally, although identified as a potential issue, there 
ihave been no studies to delineate exposure of on-site personnel through direct contact with | 
iiboth the soil and the near surface ground water, which is likely connected to the San 
{Francisco Bay. In turn, this presents the further issues of migrating contaminates. Lastly, 
ithe soil supporting the Island (TI) is man-made composed of dredged materials consisting 
lof poorly graded "fine" sand placed over Yerba Buena Shoals. This will allow for the 
ontaminates to seap through to the water. Additionally, although YBI is a natural rock 
island with minimal soil cover, there is still the issue of runoff contaminates into the water. | 
It is foreseeable that the contaminates from both of these areas would transport potential —j 
ontaminants to the Bay or runoff would infiltrate into the Franciscan sandstone and shale. 
{}Concern 2: It is known that this area hosts migratory birds, mammals, and fish. These . 
| include waterfowl and passerine birds, the California sea lion and harbor seal. A survey of | 
both Federal and California endangered or threatened species identified seven animals and 
17 plant species in the contaminated area. | 
{jOverall: There have been no studies showing what the effects of how the removal from one 
i site will effect the water and soil. Where will it be removed to? stored? Process? effect on 
ildlife? Due to the ongoing possible contamination of the water, it would be more 
|Jamenable to the community to have the entire site cleaned. This would cause less disruption 
to the wildlife and it would ensure as little further contamination of the water as possible. _ 
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